Temporal processes for aggregating multi dimensional data from discrete and distributed collectors to provide enhanced space-time perspective

ABSTRACT

At a central controlling system, a composite map is constructed based on a previous dataset that includes (A) images of a target obtained by imaging devices at a first time, and (B) respective meta data associated with the images. Imaging devices collect image data during a second time after the first time. In accordance with the composite map, the imaging devices obtain images of the target and associate meta data with the images. The images and respective meta data are communicated to the central controlling system. The positions and orientations of the imaging devices when the respective images were obtained, and when the images were obtained, are used to index the images against the target, thereby aggregating multi-dimensional data for the target. Temporal information about a characteristic of the target over time is then extracted from the aggregated multi-dimensional data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a National Stage of International Patent ApplicationNo. PCT/US2015/057421 filed Oct. 26, 2015 and published as WO2016/069498 A1, which claims priority to U.S. Provisional ApplicationNo. 62/209,787 filed Aug. 25, 2015, U.S. Provisional Application No.62/206,754 filed Aug. 18, 2015, U.S. Provisional Application No.62/203,310 filed Aug. 10, 2015, and U.S. Provisional Application No.62/068,728 filed Oct. 26, 2014, the entire contents of whichapplications is incorporated herein for all purposes by this reference.

TECHNICAL FIELD

This relates generally to image processing and informatics, includingbut not limited to capturing and consolidating image data using multiplesensor devices and dynamic control signals.

BACKGROUND

The use of imaging technology for analyzing and detecting changes ofsurface structures has a number of broad biomedical and non-biologicalapplications, ranging from medical imaging and disease detection, toverifying the integrity of building structures. Despite significantadvances in the processing and imaging capabilities of consumer devices,imaging technology and equipment enabling this surface imaging andanalysis functionality has traditionally been prohibitively costly andimpractical for adoption by the broad consumer demographic. Furthermore,mechanisms for aggregating and aligning subject data on a large scalefor enhanced surface informatics based detection also remainsubstantially undeveloped.

SUMMARY

Accordingly, there is a need for faster, more efficient methods,systems, devices, and interfaces for capturing, aligning, andaggregating image data at different times using sensor devices. Imagesand associated meta data of a particular subject may be captured atdifferent times by utilizing the robust sensing capabilities of sensordevices, such as smart phones equipped with cameras, accelerometers, andgyroscopes. By using information from previous data captures, such aspositional and orientation data associated with images from previousdata captures, subsequent data may be obtained in an aligned fashion,thereby permitting captured data to be aggregated and temporallyanalyzed. Such methods and interfaces optionally complement or replaceconventional methods for capturing, aligning, and aggregating image datausing sensor devices.

In accordance with some embodiments, a method is performed at a centralcontrolling system (e.g., a processing device) having one or moreprocessors and memory for storing one or more programs for execution bythe one or more processors. The method includes constructing a compositemap based on a previous dataset. The previous dataset includes (A) aplurality of two-dimensional pixilated images of a target obtained byone or more computer-enabled imaging devices (e.g., client devices, suchas smart phones) during a first time, and (B) respective meta dataassociated with each of the plurality of two-dimensional pixilatedimages, wherein the respective meta data indicates (i) positional andorientation data that includes a position and orientation of arespective computer-enabled imaging device when the respective image isobtained, (ii) an indication of when the respective image is obtained,and an (iii) identity of the respective computer-enabled imaging device.Each of the one or more computer-enabled imaging devices are used tocollect respective image data of the target during a second timesubsequent to the first time, by causing each respectivecomputer-enabled imaging device of the one or more computer-enabledimaging devices to execute a method of data capture. The method of datacapture includes obtaining, in accordance with the composite map basedon the previous dataset, a respective two-dimensional pixilated image ofthe target. Meta data is associated with the respective two-dimensionalpixilated image, wherein the respective meta data indicates respective(i) positional and orientation data that includes a position andorientation of the respective computer-enabled imaging device when therespective image is obtained, (ii) an indication of when the respectiveimage is obtained, and an (iii) identity of the respectivecomputer-enabled imaging device. The respective two-dimensionalpixilated image of the target and the respective meta data are thencommunicated to the central controlling system. The central controllingsystem receives one or more respective two-dimensional pixilated imagesand associated meta data collected from the second time from each of theone or more computer-enabled imaging devices. The position andorientation of each of the respective computer-enabled imaging deviceswhen the respective images were obtained, and the indication of when theimages were obtained, are used to index the images against the target,thereby aggregating multi-dimensional data for the target. Temporalinformation about a characteristic of the target over time is thenextracted from the aggregated multi-dimensional data for the target.

In accordance with some embodiments, a computer-enabled imaging deviceincludes a processor and memory for storing one or more programs forexecution by the processor, the one or more programs includinginstructions for performing any of the operations described above.

In accordance with some embodiments, a central controlling systemincludes a processor and memory for storing one or more programs forexecution by the processor, the one or more programs includinginstructions for performing any of the operations described above.

In accordance with some embodiments, a computer-readable storage mediumstoring one or more programs for execution by one or more processors,the one or more programs including instructions for performing any ofthe operations described above.

Thus, devices and systems are provided with faster, more efficientmethods for capturing, aligning, and aggregating image data, therebyincreasing the value, effectiveness, efficiency, and user satisfactionwith such devices and systems.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings. Like reference numerals referto corresponding parts throughout the figures and description.

FIG. 1 is a block diagram illustrating an exemplary temporal imagingsystem, in accordance with some embodiments.

FIG. 2 is a block diagram illustrating an exemplary processing device,in accordance with some embodiments.

FIG. 3 is a block diagram illustrating an exemplary client device, inaccordance with some embodiments.

FIGS. 4A-4B illustrate an environment in which image data is capturedand aggregated for a target over time, in accordance with someembodiments.

FIGS. 5A-5D are flow diagrams illustrating a method for capturing andaggregating image data for a target over time, in accordance with someembodiments.

DESCRIPTION OF EMBODIMENTS

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings. In the following description,numerous specific details are set forth in order to provide anunderstanding of the various described embodiments. However, it will beapparent to one of ordinary skill in the art that the various describedembodiments may be practiced without these specific details. In otherinstances, well-known methods, procedures, components, circuits, andnetworks have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are used onlyto distinguish one element from another. For example, a first smartphone could be termed a second smart phone, and, similarly, a secondsmart phone could be termed a first smart phone, without departing fromthe scope of the various described embodiments. The first smart phoneand the second smart phone are both smart phones, but they are not thesame smart phone.

The terminology used in the description of the various embodimentsdescribed herein is for the purpose of describing particular embodimentsonly and is not intended to be limiting. As used in the description ofthe various described embodiments and the appended claims, the singularforms “a,” “an,” and “the” are intended to include the plural forms aswell, unless the context clearly indicates otherwise. It will also beunderstood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“includes,” “including,” “comprises,” and/or “comprising,” when used inthis specification, specify the presence of stated features, integers,steps, operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

As used herein, the term “exemplary” is used in the sense of “serving asan example, instance, or illustration” and not in the sense of“representing the best of its kind.”

FIG. 1 is a block diagram illustrating a temporal imaging system 100, inaccordance with some embodiments. The imaging system 100 includes anumber of client devices (also called “computer-enabled imagingdevices,” “client systems,” “client computers,” or “clients”) 104-1,104-2, 104-3 . . . 104-n and a processing device 108 (also called a“central controlling system”) communicably connected to one another byone or more networks 106 (e.g., the Internet, cellular telephonenetworks, mobile data networks, other wide area networks, local areanetworks, metropolitan area networks, and so on).

In some embodiments, the one or more networks 106 include a publiccommunication network (e.g., the Internet and/or a cellular datanetwork), a private communications network (e.g., a private LAN orleased lines), or a combination of such communication networks. In someembodiments, the one or more networks 106 use the HyperText TransportProtocol (HTTP) and the Transmission Control Protocol/Internet Protocol(TCP/IP) to transmit information between devices or systems. HTTPpermits client devices to access various resources available via the oneor more networks 106. In some embodiments, the one or more networks 106are wireless communications channels based on various custom or standardwireless communications protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee,6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART,MiWi, etc.), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument. Alternatively, in some embodiments, at least a portion of theone or more networks 106 comprise physical interfaces based on wiredcommunications protocols (e.g., Ethernet, USB, etc.). Although alldevices are illustrated as being interconnected through the network 106,in some implementations, any of the aforementioned devices or systemsare communicably connected with each other either directly (i.e.,device-to-device) or through a network device (e.g., a routerrepresented by network 106), or with only a subset of the other devicesof the imaging system 100, via any combination of the aforementionednetworks 106 (e.g., client devices 104 communicate with one another viaBluetooth, transmit time-stamped images to the processing device 108 viaa cellular network, and receive control signals from the control device110 via Wi-Fi). The various embodiments of the invention, however, arenot limited to the use of any particular communication protocol.

In some embodiments, the client devices 104-1, 104-2, . . . 104-n arecomputing devices such as cameras, video recording devices, smartwatches, personal digital assistants, portable media players, smartphones, tablet computers, 2D devices, 3D (e.g., virtual reality)devices, laptop computers, desktop computers, televisions with one ormore processors embedded therein or coupled thereto, in-vehicleinformation systems (e.g., an in-car computer system that providesnavigation, entertainment, and/or other information), and/or otherappropriate computing devices that can be used to capture various typesof data (e.g., multimedia, such as image, video, and/or audio data; metadata; etc.), as well as communicate with other client devices 104 and/orthe processing device 108.

In some embodiments, client devices are configured to be mounted on orattached to various apparatuses/platforms which affect and dictate amotion of the client device during data capture. Client devices may, forexample, be fixed to structures (e.g., walls, ceilings), attached tovehicles (e.g., bikes, automobiles, planes, drones, etc.), and/orattached to humans/animals (e.g., via clothing, helmets, collars) torecord subjects or activities in a multidimensional manner (e.g.,spatially and temporally). In some embodiments, mobile apparatuses towhich client devices are mounted include one or more processors andmemory storing instructions (e.g., received control signals includingpositional/orientation data from a previous dataset/composite map;instructions which, when executed, actuate mechanical components of themobile apparatus such that the mobile apparatus and the mounted clientdevice are aligned with a specified position and/or orientation) forexecution by the one or more processors. In some embodiments, mobileapparatuses include at least some of the same operational capabilitiesand features of the client devices 104, which may be used additionally,alternatively, and/or in conjunction with the client devices 104 (e.g.,drone devices include additional sensors that may be used in conjunctionwith sensors of the client devices 104). In some embodiments, the firstclient device is fixedly mounted to the mobile apparatus (e.g., drone)such that sensor readings by the first client device are substantiallyrepresentative of environmental conditions associated with the mobileapparatus. For example, sensor readings obtained by the first clientdevice that indicate an orientation of the first client device, alsoindicate an orientation of a mobile apparatus to which the first clientdevice is mounted. In other words, in some embodiments, because thefirst client device and the mobile apparatus are fixedly mounted, theirrespective orientations are substantially the same. Similarly, asanother example, a location of the first client device (derived fromsensor readings acquired by the first client device) is substantiallythe same as a location of the mobile apparatus.

Client devices 104 (which may be mounted to respective mobileapparatuses) may be deployed to obtain or generate data for a designatedtarget (e.g., facial region of a human subject, crop fields, urbanlandscapes, etc.) for later processing and analysis (e.g., transmittingcaptured data to a processing device 108 and/or other client devices forprocessing). Client devices 104 may also be configured to receive,display, and/or manipulate data (e.g., data generated, obtained, orproduced on the device itself; constructed composite maps; aggregatedmulti-dimensional data received from the processing device 108 or otherclient devices, etc.). In some embodiments, the client devices 104(and/or respective mobile apparatuses) capture multimedia data (e.g.,time-stamped images, video, audio, etc.), and associate respective metadata (e.g., environmental information (time, geographic location),device readings (sensor readings from accelerometers, gyroscopes,barometers), etc.) with the captured multimedia data. After the captureddata is processed (e.g., by a processing device 108, client devices 104,etc.), the same or other client devices 104 may subsequently receivedata from the processing device 108 and/or other client devices fordisplay (e.g., temporally, spectrally, and/or spatially aggregatedmulti-dimensional data, including two or three-dimensional compositemaps, point clouds, textured maps, etc.).

The processing device 108 (which, in some embodiments, may itself be aclient device 104) stores, processes, aggregated, consolidates, and/oranalyzes data received from one or more devices (e.g., datasets of asubject received from client devices 104, which include multimedia data,associated meta data, etc.). The resulting data of such processing andanalysis (e.g., constructed composite maps, aggregated multi-dimensionaldata) are in turn disseminated to the same and/or other devices forviewing, manipulation, and/or further processing and analysis. In someembodiments, the processing device 108 consolidates data received fromone or more client devices 104 and performs one or more geomatics basedprocesses. For example, using associated meta data, the processingdevice 108 constructs two or three-dimensional composite maps, wherecollection of image data at subsequent times is performed in accordancewith the constructed composite map (e.g., aligning the positions andorientations of client devices with positional/orientation data ofpreviously captured images in order to enable temporal analyses). Insome embodiments, useful biological or non-biological data is furtherderived and extracted from visual representations generated by geomaticsbased processes (e.g., extracting temporal data the aggregatedmulti-dimensional data). Extracted data can be further processed oranalyzed for detection purposes (e.g., detecting a temporally observablechange). In some embodiments, the processing device 108 is a singlecomputing device such as a computer server, while in other embodiments,the processing device 108 is implemented by multiple computing devicesworking together to perform the actions of a server system (e.g., cloudcomputing).

In some embodiments, the processing device 108 (or one or more clientdevices 104) also serves as a control device for guiding subsequentcollection of image data by one or more devices. The processing devicemay, for example, send control signals/commands to one or more devices(e.g., client device 104, mobile apparatus) for execution duringsubsequent image collection, where the control signals may includeinstructions executable by the receiving devices that modify parametersof a mobile pattern (e.g., the positioning of a drone) or captureparameters (e.g., increased image resolution, data capture start/endtime, capture schedule, position/orientation of capture, etc.) of thereceiving devices.

In some embodiments, data (e.g., aggregated multi-dimensional data) issent to and viewed by client devices in a variety of output formats,and/or for further processing or manipulation (e.g., CAD programs, 3Dprinting, virtual reality displays, holography applications, etc.). Insome embodiments, data is sent for display to the same client devicethat performs the image capture and acquires sensor readings (e.g.,client devices 104), and/or other systems and devices (e.g., dataapparatus 108, a client device 104-3 that is a dedicated viewingterminal, etc.). In some embodiments, client devices 104 access dataand/or services provided by the processing device 108 by execution ofvarious applications. As another example, one or more of the clientdevices 104-1, 104-2, . . . 104-n execute software applications that arespecific to viewing and manipulating data (e.g., surface informatics“apps” running on smart phones or tablets).

FIG. 2 is a block diagram illustrating an exemplary processing device108, in accordance with some embodiments. In some embodiments, theprocessing device 108 is a central controlling system, client device(e.g., one or more client devices 104, FIG. 1), processing deviceapparatus, server system, or any other electronic device for receiving,collecting, storing, consolidating, displaying, and/or processing datareceived from a plurality of devices over a network (sometimes referredto alternatively as a data processing and display system).

The processing device 108 typically includes one or more processingunits (processors or cores) 202, one or more network or othercommunications interfaces 204, memory 206, and one or more communicationbuses 208 for interconnecting these components. The communication buses208 optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components. Theprocessing device 108 optionally includes a user interface (not shown).The user interface, if provided, may include a display device andoptionally includes inputs such as a keyboard, mouse, trackpad, and/orinput buttons. Alternatively or in addition, the display device includesa touch-sensitive surface, in which case the display is atouch-sensitive display.

Memory 206 includes high-speed random-access memory, such as DRAM, SRAM,DDR RAM, or other random-access solid-state memory devices; and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, and/orother non-volatile solid-state storage devices. Memory 206 optionallyincludes one or more storage devices remotely located from theprocessor(s) 202. Memory 206, or alternately the non-volatile memorydevice(s) within memory 206, includes a non-transitory computer-readablestorage medium. In some embodiments, memory 206 or the computer-readablestorage medium of memory 206 stores the following programs, modules anddata structures, or a subset or superset thereof:

-   -   an operating system 210 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 212 that is used for connecting        the processing device 108 to other computers, systems, and/or        client devices 104 via the one or more communication network        interfaces 204 (wired or wireless) and one or more communication        networks (e.g., the one or more networks 106)    -   a subject data store 214 for storing captured data associated        with a target (e.g., captured by one or more client devices 104,        FIGS. 1 and 3), such as:        -   multimedia data 2140 for storing multimedia data (e.g.,            time-stamped images, video, audio, etc.) captured by one or            more sensors or devices (e.g., two-dimensional pixilated            detector and/or microphone of a client device 104, FIG. 3)            of the client devices 104 (and/or mobile apparatuses); and        -   meta data 2142 for storing meta data (e.g., device data,            environmental device measurements, and/or other data            associated with captured multimedia) acquired by a client            device 104 (and/or respective mobile apparatuses), including            but not limited to: device identifiers (e.g., identifying            the device of a group of devices that captured the            multimedia item, which may include an arbitrary identifier,            a MAC address, a device serial number, etc.), temporal data            (e.g., date and time of a corresponding capture), location            data (e.g., GPS coordinates of the location at which            multimedia item was captured), multimedia capture/device            settings (e.g., image resolution, focal length, frequency at            which images are captured, frequency ranges that a pixilated            detector is configured to detect, etc.), sensor frequencies            (e.g., the respective frequency at which sensors of a device            captured data, such as an accelerometer frequency, a            gyroscope frequency, a barometer frequency, etc.),            accelerometer readings (e.g., in meters/sec2), positional            data (e.g., (x, y, z) coordinates of the device with respect            to a pre-defined axes or point of reference), orientation            data (e.g., roll (ϕ), pitch (θ), yaw (ψ)), and/or any            additional sensor or device measurements or readings for            determining spatial, spectral, and/or temporal            characteristics of a device, targets/subjects, or regions of            interest;    -   geomatics module 216 for processing, manipulating, aggregating,        and analyzing data (e.g., image data and associated meta data        received from one or more client devices 104) in order to        generate and view spatial, spectral, and/or temporal        representations of a target (e.g., constructed composite maps,        aggregated multi-dimensional data, etc.);    -   processing module 218 for processing, analyzing, and extracting        data (e.g., temporal data from) from generated spatial,        spectral, and/or temporal representations of the target (e.g.,        constructed maps, aggregated multi-dimensional data, etc.), for        detecting temporal observable changes and/or conditions (e.g.,        determining satisfaction of change thresholds, etc.), and/or for        generating control signals that cause collection of image data        (e.g., sent to one or more client device 104); and    -   dissemination module 220 for sending data (e.g., constructed        composite maps, aggregated multi-dimensional data, alerts, etc.)        for viewing and/or further processing.

The subject data store 214 (and any other data storage modules) storesdata associated with one or more subjects in one or more types ofdatabases, such as graph, dimensional, flat, hierarchical, network,object-oriented, relational, and/or XML databases, or other data storageconstructs.

FIG. 3 is a block diagram illustrating an exemplary client device 104,in accordance with some embodiments.

The client device 104 (e.g., a computer-enabled imaging device, such asa smart phone) typically includes one or more processing units(processors or cores) 302, one or more network or other communicationsinterfaces 304, memory 306, and one or more communication buses 308 forinterconnecting these components. The communication buses 308 optionallyinclude circuitry (sometimes called a chipset) that interconnects andcontrols communications between system components. The client device 104includes a user interface 310. The user interface 310 typically includesa display device 312. In some embodiments, the client device 104includes inputs such as a keyboard, mouse, and/or other input buttons316. Alternatively or in addition, in some embodiments, the displaydevice 312 includes a touch-sensitive surface 314, in which case thedisplay device 312 is a touch-sensitive display. In client devices thathave a touch-sensitive display 312, a physical keyboard is optional(e.g., a soft keyboard may be displayed when keyboard entry is needed).The user interface 310 also includes an audio output device 318, such asspeakers or an audio output connection connected to speakers, earphones,or headphones. Furthermore, some client devices 104 use a microphone andvoice recognition to supplement or replace the keyboard. Optionally, theclient device 104 includes an audio input device 320 (e.g., amicrophone) to capture audio (e.g., speech from a user). Optionally, theclient device 104 includes a location detection device 322, such as aGPS (global positioning satellite) or other geo-location receiver, fordetermining the location of the client device 104.

The client device 104 also optionally includes an image/video capturedevice 324, such as a camera or webcam. In some embodiments, theimage/video capture device 324 includes a two-dimensional pixilateddetector/image sensor configured to capture images at one or morepredefined resolutions (e.g., a low resolution, such as 480×360, and ahigh resolution, such as 3264×2448). In some embodiments, theimage/video capture device 324 captures a plurality of images (e.g., astream of multiple images) at a predefined frequency (e.g., 30 Hz). Insome embodiments, the client device 104 includes a plurality ofimage/video capture devices 324 (e.g., a front facing camera and a backfacing camera), where in some implementations, each of the multipleimage/video capture devices 324 captures distinct images for subsequentprocessing (e.g., capturing images at different resolutions, ranges oflight, etc.). Optionally, the client device 104 includes one or moreilluminators (e.g., a light emitting diode) configured to illuminate atarget (e.g., subject, environment, etc.). In some embodiments, the oneor more illuminators are configured to illuminate specific wavelengthsof light (e.g., ultraviolet, infrared, polarized, fluorescence, fornight time operations when there is less than a threshold level ofambient light, for example), and the image/video capture device 324includes a two-dimensional pixilated detector/image sensor configuredwith respect to wavelength(s) of the illuminated light. Additionallyand/or alternatively, the image/video capture device 324 includes one ormore filters configured with respect to wavelength(s) of the illuminatedlight (i.e., configured to selectively filter out wavelengths outsidethe range of the illuminated light).

In some embodiments, the client device 104 includes one or more sensors326 including, but not limited to, accelerometers, gyroscopes,compasses, magnetometer, light sensors, near field communicationtransceivers, barometers, humidity sensors, temperature sensors,proximity sensors, lasers, range finders (e.g., laser-based), and/orother sensors/devices for sensing and measuring various environmentalconditions. In some embodiments, the one or more sensors operate andobtain measurements at respective predefined frequencies.

Memory 306 includes high-speed random-access memory, such as DRAM, SRAM,DDR RAM or other random-access solid-state memory devices; and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid-state storage devices. Memory 306 may optionallyinclude one or more storage devices remotely located from theprocessor(s) 302. Memory 306, or alternately the non-volatile memorydevice(s) within memory 306, includes a non-transitory computer-readablestorage medium. In some embodiments, memory 306 or the computer-readablestorage medium of memory 306 stores the following programs, modules anddata structures, or a subset or superset thereof:

-   -   an operating system 328 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks, and doing so in accordance with one or more        control signals (e.g., operating the image/video capture module        332/sensor module 338 in accordance with received control        signals from the processing device 108);    -   a network communication module 330 that is used for connecting        the client device 104 to other computers, systems (e.g.,        processing device 108), control devices (e.g., control device        110), client devices 104, and/or drone device 102 via the one or        more communication network interfaces 304 (wired or wireless)        and one or more communication networks (e.g., Internet, cellular        telephone networks, mobile data networks, other wide area        networks, local area networks, metropolitan area networks, IEEE        802.15.4, Wi-Fi, Bluetooth, etc.);    -   an image/video capture module 332 (e.g., a camera module) for        processing a respective image or video captured by the        image/video capture device 324, where the respective image or        video may be sent or streamed (e.g., by a client application        module 340) to the processing device 108;    -   an audio input module 334 (e.g., a microphone module) for        processing audio captured by the audio input device 320, where        the respective audio may be sent or streamed (e.g., by a client        application module 340) to the processing device 108;    -   a location detection module 336 (e.g., a GPS, Wi-Fi, or hybrid        positioning module) for determining the location of the client        device 104 (e.g., using the location detection device 322) and        providing this location information for use in various        applications (e.g., client application module 340);    -   a sensor module 338 for obtaining, processing, and transmitting        meta data (e.g., device data, environmental device measurements,        and/or other data associated with captured multimedia) acquired        by the client device 104 and/or a respective drone device 102,        including but not limited to: device identifiers (e.g.,        identifying the device of a group of devices that captured the        multimedia item, which may include an arbitrary identifier, a        MAC address, a device serial number, etc.), temporal data (e.g.,        date and time of a corresponding capture), location data (e.g.,        GPS coordinates of the location at which multimedia item was        captured), multimedia capture/device settings (e.g., image        resolution, focal length, frequency at which images are        captured, frequency ranges that a pixilated detector is        configured to detect, etc.), sensor frequencies (e.g., the        respective frequency at which sensors of a device captured data,        such as an accelerometer frequency, a gyroscope frequency, a        barometer frequency, etc.), accelerometer readings (e.g., in        meters/sect), positional data (e.g., (x, y, z) coordinates of        the device with respect to a pre-defined axes or point of        reference), orientation data (e.g., roll (ϕ), pitch (θ), yaw        (ω)), and/or any additional sensor or device measurements or        readings for determining spatial, spectral, and/or temporal        characteristics of a device, targets/subjects, or regions of        interest; and    -   one or more client application modules 340, including the        following modules (or sets of instructions), or a subset or        superset thereof:        -   a control module for receiving (e.g., from a processing            device 108, client device 104, etc.), generating (e.g.,            based on a composite map), storing, providing,            re-broadcasting, and/or operating components of the client            device 104 in accordance with control signals (e.g.,            received from processing device 108) and/or composite maps            (e.g., meta data of datasets used to construct a composite            map, such as positional/orientation data associated with            images of the datasets);        -   a web browser module (e.g., Internet Explorer by Microsoft,            Firefox by Mozilla, Safari by Apple, or Chrome by Google)            for accessing, viewing, and interacting with web sites            (e.g., a web site provided by the processing device 108),            captured data (e.g., images), and/or other received data            (e.g., composite map, aggregated multi-dimensional data,            etc.); and/or        -   other optional client application modules for viewing and/or            manipulating captured data or other received data, such as            applications for photo management, video management, a            digital video player, computer-aided design (CAD), 3D            viewing (e.g., virtual reality), 3D printing, holography,            and/or other graphics-based applications.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functions asdescribed above and/or in the methods described in this application(e.g., the computer-implemented methods and other information processingmethods described herein). These modules (i.e., sets of instructions)need not be implemented as separate software programs, procedures ormodules, and thus various subsets of these modules are, optionally,combined or otherwise re-arranged in various embodiments. In someembodiments, memory 206 and/or 306 store a subset of the modules anddata structures identified above. Furthermore, memory 206 and/or 306optionally store additional modules and data structures not describedabove.

Furthermore, in some implementations, the functions of any of thedevices and systems described herein (e.g., client devices 104,processing device 108, etc.) are interchangeable with one another andmay be performed by any other devices or systems, where thecorresponding sub-modules of these functions may additionally and/oralternatively be located within and executed by any of the devices andsystems. As one example, although the client device 104 (FIG. 3)includes sensors and modules for obtaining/processing images (e.g.,sensors 326 and an image/video capture module 332) and obtainingrespective sets of meta data (e.g., sensor module 338) in accordancewith a composite map, in some embodiments a mobile apparatus to whichthe client device 104 is mounted (not illustrated) may include analogousmodules, components, and device capabilities for performing the sameoperations (e.g., sensors and modules containing instructions forobtaining images and respective meta data in accordance with thecomposite map). The devices and systems shown in and described withrespect to FIGS. 1 through 3 are merely illustrative, and differentconfigurations of the modules for implementing the functions describedherein are possible in various implementations.

FIGS. 4A-4B illustrate an environment in which image data is capturedand aggregated for a target over time, in accordance with someembodiments.

Specifically, the environment shown in FIG. 4A includes client devices104-1 through 104-4 for obtaining, during a first time T1, a datasetthat includes multiple images and respective meta data of a target(e.g., user 102-1) in order to construct a composite map. Continuingwith the example in FIG. 4B, the composite map, constructed from theprevious dataset, is subsequently used by the same client device 104-1at a later time T2 to collect additional image data and meta data of thetarget. In particular, the client device 104-1 is positioned andoriented based on meta data of the previous dataset such thatadditionally captured image data during time T2 is aligned with theprevious dataset. The additional image data and respective meta data, incombination with the previous dataset, are received at a centralcontrolling system for aggregation as multi-dimensional data for thetarget. Temporal information about characteristics of the target maythen be extracted and analyzed. Although the client devices 104 aresmart phones in the example illustrated, in other implementations theclient devices 104 may be any electronic device with image capturecapabilities (e.g., a camera, a PDA, etc.). Furthermore, while thetarget is a live, biological subject (e.g., a human), the illustratedenvironment and processes described herein are also applicable tonon-biological contexts (as described in greater detail with respect toFIGS. 5A-5D).

Referring to FIG. 4A, the client devices 104 are used to capture one ormore still-frame images, video sequences, and/or audio recordings fromone or more positions and orientations during time T1. Time T1 (FIG. 4A)and time T2 (FIG. 4B) may correspond to specific intervals (e.g., 20seconds) or moments of time (e.g., dates, days, hours, seconds, etc.).Concurrently with image capture, client devices 104 also acquire andassociate meta data with the obtained images. Meta data includestemporal information (e.g., indication of time of image capture), deviceinformation (e.g., identity of a respective client device, such as aunique device identifier), and sensor readings of various environmentalconditions obtained from one or more sensors of the client device 104-1(e.g., sensors 326, such as an accelerometer, gyroscope, barometer,etc.) from which positional and orientation data for a client device canbe determined. Positions of the client devices 104 may be defined withrespect to a distance from the target (e.g., distance d₁ of the clientdevice 104-1), and/or coordinates within a predefined coordinate system(e.g., (x, y, z) coordinates based on a coordinate system defined by thesubject top view and a height of the target). An orientation of theclient devices 104 is measured with respect to a reference orientation.In this example, orientations of the client devices are defined by anangle of rotation within the x-y axis (i.e., yaw (ψ)), an angle ofrotation within the y-z axis (i.e., pitch (θ)), and an angle of rotationwithin the x-z axis (i.e., roll (ϕ). Meta data is also based on andincludes various time-stamped sensor readings obtained from one or moresensors of the client device 104-1 (e.g., sensors 326, such as anaccelerometer, gyroscope, barometer, etc.). Other types of meta data aredescribed in greater detail throughout. In this example, image and metadata are captured in the medical context of analyzing a subject's skincondition. As shown on the screen of client device 104-1, a lesion 402is observed in the facial region of user 102-1.

After data capture during time T1, the captured image data and meta datacomprise a collective dataset that is used to construct a two orthree-dimensional composite map (not shown). The map, which may be aspatial, spectral, and/or temporal representation of the target, issaved and subsequently used in later data capture processes. Techniquesfor constructing and using the composite map are described in greaterdetail with respect to FIG. 5A-5D.

Referring now to FIG. 4B, the composite map previously constructed basedon the dataset obtained in FIG. 4A is used to guide subsequent datacapture during a later time T2. Specifically, the composite map may beused to align subsequently captured images of a target or a particularregion of the target such that temporal information may be extracted.

Continuing the example above, a temporal analysis is performed withrespect to the lesion 402 as observed from the particular position andorientation of the client device 104-1 during time T1. As shown in FIG.4B, prior to time T2, the client device 104-1 is not initially alignedwith the position and orientation of the client device 104-1 at time T1,having a position to the right-hand side of the user 102-1. Using theconstructed composite map, and deriving the positional and orientationdata associated with the images of the previous dataset corresponding toimages captured by the client device 104-1 during time T1, the clientdevice 104-1 is guided (e.g., by providing control signals to a mobileapparatus to which the client device 104-1 is mounted, displayinginstructions for repositioning/reorienting, etc.) into a position suchthat aligned image and meta data can be captured for temporal analysis.That is, by using positional and orientation data (e.g., (x, y, z)coordinates, distance d₁, ψ₁) to align the client device 104-1 at timeT2 with the client device 104-1 from time T1, subsequent image and metadata obtained during time T2 are aligned with the data captured in T1,and temporal changes may be observed with respect to the lesion 402.Here, the lesion 402 exhibits growth between times T1 and T2.

After data collection concludes, image and meta data captured duringtime T1 (e.g., for constructing the composite map) and time T2 (e.g.,for the observed lesion 402) is received by a central controlling system(e.g., processing device 108, FIG. 1) and aggregated. Aggregation may,for example, include building upon the existing composite map in orderto further refine and add points to the composite map. Temporalinformation may thereafter be extracted from the aggregatedmulti-dimensional data, and further processed and/or displayed in orderto render analyses of a target or feature of interest over time. Dataaggregation, and further analytical operations and processes, aredescribed in greater detail with respect to FIG. 5A-5D.

FIGS. 5A-5D are flow diagrams illustrating a method 500 for capturingand aggregating image data for a target over time, in accordance withsome embodiments. In some implementations, the method 500 is performedby one or more devices of one or more systems (e.g., client devices 104,processing device 108, etc. of a temporal imaging system 100, FIGS.1-3), or any combination thereof. Thus, in some implementations, theoperations of the method 500 described herein are entirelyinterchangeable, and respective operations of the method 500 areperformed by any one of the aforementioned devices and systems, orcombination of devices and systems. For ease of reference, the methodsherein will be described as being performed by a processing device(e.g., processing device 108, FIGS. 1 and 2) or one or more clientdevices (e.g., client devices 104, FIGS. 1, 3, 4A-4B). While parts ofthe methods are described with respect to a processing device or clientdevice, any operations or combination of operations of the method 500may be performed by any electronic device having imagecapture/processing capabilities (e.g., a computer-enabled imagingdevice, such as a smart phone, a camera device, a computer-enabledimaging device, a PDA, etc.; a central controlling system/server; etc.).Steps of the method 500 correspond to instructions/programs stored in amemory or other computer-readable storage medium of a processing device(e.g., memory 206 of processing device 108, FIG. 2), theinstructions/programs for execution by one or more processors of theprocessing device (e.g., 202). Additionally and/or alternatively, stepsof the method 500 correspond to instructions/programs stored in a memoryor other computer-readable storage medium of a client device (e.g.,memory 306 of client device 104, FIG. 3), the instructions/programs forexecution by one or more processors of the processing device (e.g.,302).

Although some steps of the method 500 are described with respect toeither a first or second client device of a plurality of client devices,any operations performed by the second client device may be performed inaccordance with any of the embodiments described with respect to thefirst client device, and vice versa. Furthermore, any respectiveoperations performed by the first and/or second client device may beperformed additionally, alternatively, and/or concurrently with oneanother (e.g., concurrent obtaining of workflows). Moreover, anyoperations described with respect to the first and/or second clientdevice may be analogously performed by one or more additional clientdevices of the temporal imaging system 100 (or other devices/systemsdescribed herein, such as an additional mobile apparatuses),additionally, alternatively, and/or concurrently with the operations ofthe first and/or second client device.

As an overview of the method 500, in some embodiments, a processingdevice (e.g., processing device 108, FIG. 1) constructs a composite map(502, FIG. 5A) based on a previous dataset. The previous datasetincludes: (A) a plurality of two-dimensional pixilated images of thetarget obtained by one or more computer-enabled imaging devices (e.g.,client device 104) during a first time, and (B) respective meta dataassociated with each of the plurality of two-dimensional pixilatedimages. Each of the one or more client devices are used (518, FIG. 5B)to collect respective image data of the target during a second timesubsequent to the first time, by causing each respective client deviceof the one or more client devices to execute a method of data capture.The method of data capture includes the one or more computer-enabledimaging devices obtaining (520), in accordance with the composite mapbased on the previous dataset, a respective two-dimensional pixilatedimage of the target, and associating (530, FIG. 5C) meta data with therespective two-dimensional pixilated image. The respectivetwo-dimensional pixilated image of the target and the respective metadata are communicated (532) to the processing device. The processingdevice receives (536) one or more respective two-dimensional pixilatedimages from each of the one or more client devices. The position andorientation of each of the respective client devices when the respectiveimages were obtained, and the indication of when the images wereobtained, are then used (538) to index the images against the target,thereby aggregating multi-dimensional data for the target. Temporalinformation about a characteristic of the target over time is extracted(546, FIG. 5D) from the aggregated multi-dimensional data. An exemplaryenvironment in which the method 500 is performed is described withrespect to FIGS. 4A-4B. Various embodiments of the method 500 aredescribed in greater detail below.

Referring now to FIG. 5A, the processing device (e.g., processing device108, FIG. 1) constructs (502) a composite map based on a previousdataset. The previous dataset includes: (A) a plurality oftwo-dimensional pixilated images of a target obtained by one or moreclient devices during a first time; and (B) respective meta dataassociated with each of the plurality of two-dimensional pixilatedimages, wherein the respective meta data indicates (i) positional andorientation data that includes a position and orientation of arespective client device when the respective image is obtained, (ii) anindication of when the respective image is obtained, and an (iii)identity of the respective client device (other examples of associatedmeta data are described throughout, in FIG. 3, for example). As referredto throughout, obtaining images and respective meta data for a giveninterval of time is generally referred to as an act of data capture or adata capture session. An exemplary data capture session for the previousdataset is illustrated and described with respect to FIG. 4A, whereclient devices 104 capture multiple images of a facial region of theuser 102-1 during time T1.

The composite map may be a multidimensional spatial, spectral, and/ortemporal representation of a target, where a target may be a region ofinterest (e.g., portion, feature, etc.), or any viewable aspect, of abiological (e.g., a human, crop field, etc.) or non-biological (e.g.,building surface, urban landscapes, environmental region, etc.) subject.In some embodiments, the target is the biological/non-biological subjectitself. In some embodiments, the constructed composite map represents acoordinate system (e.g., two or three-dimensional) predefined withrespect to the target. The composite map is constructed, saved, andsubsequently used in later data capture processes, and may also be usedas a basis for the generation of additional visual representations ofthe target (e.g., dense point clouds, textured meshes, etc.).

In some embodiments, constructing (502) the map includes extracting(504) two-dimensional features from each image of the previous dataset.Using the two-dimensional features extracted from the previous dataset,a three-dimensional composite map comprising a plurality ofthree-dimensional points is created (506). Each respectivethree-dimensional point of the plurality of three-dimensional points isassociated with one of a plurality of respective sets of thetwo-dimensional features extracted from the previous dataset. Each ofthe respective sets represents an appearance of an associated respectivethree-dimensional point across the previous dataset. Two-dimensionalfeatures may correspond, for example, to facial features of a human(e.g., eyes, nose, etc.), regions (e.g., particular areas of a cropfield), pixels/groups of pixels across a plurality of images, and/orother observable aspects of image data. The three-dimensional compositemap thus comprises a plurality of feature points mapped to a predefinedcoordinate system (e.g., two or three-dimensional).

In some embodiments, a two-dimensional feature is matched (508) in afirst image and in a second image of the plurality of two-dimensionalpixilated images. A parallax is then estimated (510) between the firstimage and the second image using respective meta data associated withthe first image and the second image (e.g., respective positional andorientation data of the first and second images, translational androtational trajectory derived therefrom, etc.). When the parallaxbetween the first image and the second image satisfies a parallaxthreshold, a two or three-dimensional point is added (512) to thecomposite map at a distance obtained by triangulating the first imageand the second image using respective meta data associated with thefirst image and the second image. In some embodiments, the matching(508), estimating (510), and adding (512) are repeated for differentfirst and second images of the plurality of two-dimensional pixilatedimages.

Referring now to FIG. 5B, each of the one or more client devices (e.g.,client devices 104, FIG. 4A) are used to collect respective image dataof the target during a second time subsequent to the first time, bycausing each respective client device of the one or more client devicesto execute a method of data capture. The method of data capture isdescribed in detail with respect to FIGS. 5B-5C (steps 520 through 534),and is performed by one or more client devices (e.g., client devices104-1, FIG. 4B) having respective memory storing instructions/programsfor execution by one or more respective processors of the clientdevices. In some embodiments, a subset of the one or more client devicesis used to collect respective image data of the target during the secondtime. For example, as shown in FIG. 4B, only client device 104-1 (andnot client devices 104-2 through 104-4, FIG. 4A) is used to captureimage data at the time T2. In some embodiments, a client device distinctfrom the one or more client devices is used to collect the respectiveimage data (e.g., one set of client devices 104 used to obtain imagesfor constructing the initial composite map, a different client device104 used at a later time for follow-up image capture).

In some embodiments, the processing device causes one or more clientdevices to execute the method of data capture by sending controlsignals/commands to the one or more client devices. In some embodiments,the control signals/commands include instructions executable by areceiving device (e.g., client device 104, mobile apparatus) that modifyparameters of a mobile pattern (e.g., the positioning of a drone) orcapture parameters (e.g., increased image resolution, data capturestart/end time, capture schedule, position/orientation of capture, etc.)for the receiving device. In some embodiments, the controlsignals/commands are based on and include information derived from thecomposite map or the previous dataset (e.g., associatedpositional/orientation data for a feature of interest with respect to apredefined coordinate system of the composite map).

Executing the method of data capture (518) includes obtaining (520), inaccordance with the composite map based on the previous dataset, arespective two-dimensional pixilated image of the target. In someembodiments, a plurality of two-dimensional pixilated images of thetarget is obtained. In some embodiments, multiple client devices areused to collect image data, and each of the respective two-dimensionalpixilated images are obtained in accordance with capture parameters oftheir respective client devices (e.g., position/orientation of capture,resolution, etc.), such that the captured images represent distinctaspects of the target (e.g., images obtained from distinct distances,heights, positions, orientations, etc. with respect to the target;images having distinct resolutions, capture frequencies; imagesrepresenting distinct frequencies/frequency ranges of light (UV, IR)).

In some embodiments, obtaining (520) in accordance with the compositemap includes obtaining the respective two-dimensional pixilated image inaccordance with the respective meta data of the previous dataset (i.e.,meta data associated with the plurality of two-dimensional pixilatedimages of the previous dataset, 502, FIG. 5A). Furthermore, in someembodiments, the obtaining (520) includes receiving (e.g., from theprocessing device) or identifying (e.g., from the previousdataset/composite map stored at a respective client device) at least aportion of the meta data and/or the obtained images of the previousdataset (e.g., receiving positional/orientation data or other meta data,receiving a portion of the composite map, etc.).

In some embodiments, the obtaining (520) in accordance with thecomposite map includes obtaining (522) the respective two-dimensionalpixilated image in accordance with the positional and orientation dataassociated with the plurality of two-dimensional pixilated images of theprevious dataset. Stated another way, the respective two-dimensionalpixilated image is obtained consistent with positions and/ororientations associated with the images of the previous dataset. As anexample, if the images of the previous dataset were obtained during timeT1 at a fixed orientation with respect to the target (e.g., an angle θ),the two-dimensional pixilated image obtained at time T2 is also obtainedwhile the respective client device is oriented at substantially the samefixed orientation. In some embodiments, the obtaining (522) is performedin accordance with at least a subset of the positional and orientationdata associated with the plurality of two-dimensional pixilated imagesof the previous dataset. The subset of the positional and orientationdata in the previous dataset may, for example, be a range oforientations (e.g., ψ₁<ψ<′ψ₂) with respect to, or a range of distancesfrom (e.g., d₁<d<d₂), the target. Consequently, by obtaining images inaccordance with the positional and orientation data associated with theimages of the previous dataset, image data captured at subsequent timesmay be temporally stacked and analyzed to detect changes over time.

In some embodiments, the position and orientation of the respectiveclient device, when the respective two-dimensional pixilated image isobtained, are (524) substantially aligned with the positional andorientation data associated with at least one of the plurality oftwo-dimensional pixilated images from the previous dataset. An exampleis shown FIG. 4B, where the position and orientation of client device104-1 (e.g., (x, y, z) coordinates, (ϕ, θ, ψ), etc.) are substantiallyaligned with the positional and orientation data associated with theimage captured by the client device 104-1 at time T1.

In some embodiments, the image data collected by each of the clientdevices comprises a plurality of two-dimensional pixilated imagesobtained during the second time. A position and orientation of arespective one of the one or more computer-enabled imaging devices, foreach of the plurality of two-dimensional pixilated images obtainedduring the second time, are substantially aligned with the positionaland orientation data associated with a respective one of the pluralityof two-dimensional pixilated images from the previous dataset. In otherwords, each of the images obtained during the second time have anassociated position and/or orientation that is aligned with theassociated position/orientation of an image from the previous dataset.In some embodiments, each of the images from the previous dataset withwhich the images obtained during the second time are aligned havedistinct positional and orientation data (i.e., every image obtainedduring the second time corresponds to images from the previous datasethaving associated positions/orientations that are distinct).

In some embodiments, using (518) each respective client device tocollect respective image data further includes providing informationindicating a proximity of a current position and orientation of arespective client device, to a respective position and orientationassociated with at least one of the plurality of two-dimensionalpixilated images from the previous dataset. Information indicating theproximity may include the position (e.g., (x, y, z) coordinates) and/ororientation (e.g., (ϕ, θ, ψ)) associated with at least one of the imagesfrom the previous dataset, a relative distance of the respective clientdevice to a position/orientation associated with one of the images fromthe previous dataset, and/or an image from the previous dataset (e.g.,the image shown on the screen of the client device 104-1 in FIG. 4A). Insome embodiments, providing information indicating the proximityincludes displaying, on the respective client device, visual informationindicating the proximity so as to enable alignment of the respectiveposition and orientation of the respective client device to match theposition and orientation associated with the at least one of theplurality of two-dimensional pixilated images from the previous dataset.Visual information may be a visual display of the information indicatingthe proximity (e.g., position/orientation, distance, the image, etc.).Additionally and/or alternatively, the visual information may includeon-screen instructions for guiding an associated user to reposition therespective client device so as to achieve alignment (e.g., real-timefeedback updating a proximity of the client device as it isrepositioned, provided by utilizing sensors of the client device). Forexample, referring to FIG. 4B, the position and orientation associatedwith the image captured by the client device 104-1 (FIG. 4A) isdisplayed on-screen while the client device 104-1 is in its initialposition in FIG. 4B (to the right-hand side of the user 102-1).Accordingly, a user operating the client device 104-1 may align thedevice based on the on-screen display so as to enable aligned datacapture during time T2, and further allowing for subsequent temporalanalysis. Alternatively, in some embodiments, a respective client devicereceives the information indicating the proximity of the respectiveclient device and automatically (without user intervention) alignsitself accordingly. For example, in some embodiments, the respectiveclient device receives control signals which, when executed by therespective client device (e.g., the client device is a mobile apparatus)or a mobile apparatus to which the respective client device is attached(e.g., attached drone device), aligns the respective client device.

In some embodiments, the obtaining (520) in accordance with thecomposite map includes obtaining the respective two-dimensionalpixilated image in accordance with an indication of when a respectiveimage of the previous dataset was obtained. Respective images may beobtained, for example, at the same time as the indicated time (e.g.,same time, different day), at a predefined time from the indicated time(e.g., 2 hours), or in accordance with a schedule that is based on theindicated time (e.g., same time, every other day). For example, an imagemay be obtained at a time T2 that is 2 hours from the time at which acorresponding image from the previous dataset was obtained.

In some embodiments, the first time (502, FIG. 5A, time during whichdataset is obtained for constructing the composite map) and the secondtime (518, FIG. 5B, time during which one or more client devices areused to collect image data and associate respective meta data inaccordance with the composite map) are distinct and do not overlap(e.g., the first time corresponding to a first day, the second timecorresponding to the day after). Alternatively, in some embodiments, thefirst time and the second time at least partially overlap. In someembodiments in which the first time and the second time at leastpartially overlap, data collected during the second time is added to theprevious dataset, and the composite map is further constructed/refined(e.g., add points to, refine mapping, etc.) concurrent with the datacollection at the second time.

In some embodiments, a deficiency is identified (514) (e.g.,automatically by the processing device; by a respective client device;manually by a user) in the composite map. The deficiency may correspond,for example, to a portion of the composite map (or an image of theprevious dataset) which is obstructed (e.g., object impeding view), orfor which image data is insufficient or a corresponding resolution isbelow a threshold. Furthermore, in some embodiments, the obtaining(520), for each respective client device, of the respectivetwo-dimensional pixilated image is (526) responsive to and in accordancewith the identified deficiency (e.g., additional images obtained usingcorresponding positional and/or orientation data in order to compensatethe deficiency).

In some embodiments, a two-dimensional feature of the target isidentified (516) (e.g., automatically by the processing device; by arespective client device; manually by a user) using the composite map.As described previously, two-dimensional features may correspond, forexample, to facial features of a human (e.g., eyes, nose, etc.), regions(e.g., particular areas of a crop field), pixels/groups of pixels acrossa plurality of images, and/or other observable aspects of image data(e.g., observed skin lesions, crop damage, structural damage, etc.).Furthermore, in some embodiments, the obtaining (520), for eachrespective client device, of the respective two-dimensional pixilatedimage is (528) in accordance with the identified two-dimensionalfeature. FIGS. 4A-4B illustrate such an example, where the lesion 402 isobserved in the facial region of the user 102-1 (FIG. 4A), andsubsequent image data captured by the client device 104-1 (FIG. 4B) isbased on the observed lesion 402. Additionally and/or alternatively, athree-dimensional feature of the target is identified using thecomposite map, wherein the obtaining (520), for each respective clientdevice, of the respective two-dimensional pixilated image is inaccordance with the identified three-dimensional feature.

Referring now to FIG. 5C, executing the method of data capture (518)includes associating (530) meta data with the respective two-dimensionalpixilated image, wherein the respective meta data indicates respective(i) positional and orientation data that includes a position andorientation of the respective client device when the respective image isobtained, (ii) an indication of when the respective image is obtained,and an (iii) identity of the respective client device (other examples ofassociated meta data are described throughout, in FIG. 3, for example).

Furthermore, executing the method of data capture (518) includescommunicating (532) to a processing device (or a client device) therespective two-dimensional pixilated image of the target and therespective meta data. In some embodiments, the communicating isperformed (534) wirelessly over a network (e.g., over a wirelesscommunications interface, such as IEEE 802.11 Wi-Fi, Bluetooth, etc.;using a cellular communications protocol, such as GSM, CDMA, etc.). Insome embodiments, the communicating is performed via a wired interface(e.g., transferring over a USB cable interface between the client device104-1 and the processing device 108 at the conclusion of an imagecapture session). In some embodiments, the communicating includestransferring the images to the remote processing device via a removablestorage device (e.g., a flash drive). In some embodiments, thecommunicating is performed concurrently with the obtaining (520) of therespective image and the associating (530) of respective meta data(e.g., streaming images and meta data in real-time as it is obtained).

The processing device receives (536) one or more respectivetwo-dimensional pixilated images and associated meta data collected fromthe second time from each of the one or more client devices.Furthermore, the processing device uses (538) the position andorientation of each of the respective client devices when the respectiveimages were obtained, and the indication of when the images wereobtained, to index the images against the target, thereby aggregatingmulti-dimensional data for the target. Thus, the images obtained and themeta data associated during the second time are used to temporally stackdata captured for to the target. In some embodiments, the images areindexed against the target by matching the images based on one or moretypes of associated meta data (e.g., matching based on deviceidentifier, position/orientation, time of capture, etc.).

In some embodiments, the one or more respective two-dimensionalpixilated images and associated meta data are added to the previousdataset (502), and the composite map is further constructed and/orrefined (in accordance with embodiments described with respect to FIG.5A) based on the one or more respective two-dimensional pixilated imagesand associated meta data added to the previous dataset.

Referring now to FIG. 5D, in some embodiments, the one or morerespective two-dimensional pixilated images obtained during the secondtime are spatially registered (540) using the composite map. In someembodiments, the spatial registering includes extracting (542)two-dimensional features from each image obtained during the secondtime. A set of the two-dimensional features extracted from the imagesobtained during the second time, is matched (544) with one of theplurality of respective sets of the two-dimensional features extractedfrom the previous dataset (504, FIG. 5A). The set of the two-dimensionalfeatures extracted from the images obtained during the second time isthereby associated with a respective three-dimensional point, of thethree-dimensional composite map, that is associated with the matched setof the two-dimensional features extracted from the previous dataset. Byspatially registering features extracted from images obtained duringdifferent times (e.g., the first and second times), captured image datais aligned, and subsequent temporal processing is thereafter performed(e.g., change detection for a given target or feature of a subject).

In some embodiments, at least a portion of the aggregatedmulti-dimensional data for the target is displayed (e.g., by theprocessing device, a dedicated display terminal, a client device, etc.).In some embodiments, displaying the aggregated multi-dimensional dataincludes displaying a visual indicator (e.g., time lapse video,side-by-side comparisons, composite image, etc.) for an observed changeof the target (e.g., growth of a lesion, mitigation of damage, etc.). Insome embodiments, the observed change of the target is a change observedbetween the first time (e.g., 502, FIG. 5A, when the composite map wasconstructed) and the second time (e.g., step 518, FIG. 5B, when imagedata was collected based on the composite map). In some embodiments, theaggregated multi-dimensional data is displayed with respect to aspecified time parameter (e.g., selective images for a target aredisplayed based on a predefined range of dates, a particular time,etc.). In some embodiments, the aggregated multi-dimensional data isdisplayed on a virtual reality display for display and/or manipulation.

Temporal information about a characteristic of the target over time isextracted (546) from the aggregated multi-dimensional data for thetarget. Characteristics relate to observable aspects (e.g., two orthree-dimensional physical attributes) of the target, represented byquantitative (e.g., numerical value) and/or qualitative data at orduring a particular time. As an example, characteristics of a skinlesion may be represented by data related to location (e.g., diffuse,localized), lesion size (e.g., surface area, volume), size distribution,quantity, shape, etc. Values of a characteristic may correspond totwo-dimensional (e.g., square footage of damaged crop region) orthree-dimensional measurements (e.g., volume of a blister). In contrastto values of a characteristic, temporal information represents observedchanges of a characteristic of the target over a predefined period oftime (e.g., change in value for a characteristic between a first andsecond time). For example, temporal information may indicate that sincea last image capture, the size of previously observed lesions hasincreased by a measurable amount (e.g., expressed as a quantifiableamount, such as a quantity or percentage of change). Temporalinformation may represent changes measured with respect to any specifiedpoint or range of time (e.g., difference between current data for acharacteristic and most-recently measured data, initial data, datameasured on a certain date at a certain time, etc.).

In some embodiments, extracting (546) the temporal information includesidentifying a first value for the characteristic corresponding to thefirst time, identifying a second value for the characteristiccorresponding to the second time, and comparing the first and secondvalues to determine the temporal information about the characteristic ofthe target. In some embodiments, the temporal information is adifference (i.e., a change) between the second value and the first value(e.g., change in size (surface area, volume) of a lesion over time). Insome embodiments, the temporal information is a rate of change,corresponding to a difference between the first and second values withrespect to a difference between the first and second times. That is, thetemporal information corresponds to the rate at which somecharacteristic of the measured target (e.g., volume of a blister) ischanging over a predefined period of time. The rate of change mayindicate a positive rate of change (e.g., increase of size) or anegative rate of change (e.g., shrinkage). Furthermore, in someembodiments, the temporal information is a differential rate of change(i.e., the rate at which the rate of change for a characteristic changesover time). Here, the first value is a first rate of change of thecharacteristic during the first time, the second value is a second rateof change of the characteristic during the second time, and the temporalinformation is a differential rate of change, corresponding to adifference between the first and second values with respect to adifference between the first and second times. The differential rate ofchange therefore indicates a rate at which respective rates of changefor the characteristic changes over time (e.g., the rate at whichdisease spreads through a crop field decreases over time). In someembodiments, the differential rate of change is measured with respect tovalues for a characteristic over more than two distinct points in time(e.g., a first rate of change measured between time T1 and T2, a secondrate of change measured between time T2 and T3, and a third rate ofchange measured between time T3 and T4, where the differential rate ofchange is the rate at which the rates of change measured between timesT1 and T4 change).

There are numerous examples in which the various types of temporalinformation above (e.g., change, rates of change, differential rates ofchange) may be derived. For example, changes in volume, and the rate atwhich volume changes, may be identified for lesions above the normalskin surface, blisters, edematous reactions, etc. Furthermore,continuing the example of lesion growth, extrapolated differential ratesof change may indicate an increase or decrease in cell number, wherecancerous lesions may correspond to a continuously increasing rate ofgrowth, while benign lesions correspond to a decreasing rate of growth.For lesions below the skin surface, a healing wound may correspond to adecrease in crater volume over time, or a worsening pressure ulcer maycorrespond to an increase in crater volume over time. Other examplesinclude facial swelling (e.g., change, rates of change, differentialrates of change, for the swelling of fingers/eyelids/lips over time).Within an agricultural context, rates of increasing or decreasingbiomass volume may also be measured. Another example is temporalinformation determined for surface topology (e.g., smooth to rough,multiple blisters to one coalesced blister, changes in crease structure(wrinkles, fault lines, etc.). Temporal information for changes in colordistribution include changes of characteristics (e.g., developing morepigmented globules, blood leaking into a blister, increasing greenoxidation of a copper statue), rates of change in color distribution,and differential rates of change (e.g., indicating that bleeding isslowing down, corrosion protection is effective, etc.). Temporal changesmay also be derived for three-dimensional shapes of a target (e.g.,initial spherical shape transitioning into a grape-like structure, auniform column becoming narrower at the top and thickening at the base,etc.). Changes in Tyndall effect (e.g., changes in light scatteringthrough a volume, blue scattered back to camera), or changes volume inclarity (e.g., less purulent drainage, lake clearing as runoff dirtsettles) may also be derived.

In some embodiments, temporal information is derived directly bycomparing composite maps constructed at different times in order toidentify spatial and/or spectral differences of features of the targetover time. For example, in some embodiments, the composite map based onthe previous dataset (502, FIG. 502) is a first composite map, and theimage data collected during the second time by using each of the one ormore computer-enabled imaging devices (518, FIG. 5B) includes aplurality of two-dimensional pixilated images of the target (520) andthe associated meta data (530, FIG. 5C). A second composite map isconstructed based on the image data collected during the second time.The second composite map (corresponding to the second time) is thencompared with the first composite map (corresponding to the first time)to identify respective differences in one or more three-dimensionalfeatures from the first time to the second time. In some embodiments,spatial data (e.g., positional data identified by associated meta data)of the one or more features is compared between the first and secondcomposite maps (e.g., (x, y, z) coordinates of feature points comparedin order to identify three-dimensional growth of a feature over time).In some embodiments, spectral data (e.g., color distribution of afeature) of the one or more features is compared between the first andsecond composite maps. In some embodiments, the respective identifieddifferences include a respective shape of the one or morethree-dimensional features (e.g., transition from a circular to jaggedshape). In some embodiments, the respective identified differencesinclude a respective volume of the one or more three-dimensionalfeatures. In some embodiments, the respective identified differencesinclude a respective surface contour of the one or morethree-dimensional features (e.g., a contour map is generated based ondifferences in the surface contours of a three-dimensional featurebetween the first and second times). In some embodiments, the respectiveidentified differences include a respective smoothness or roughness ofthe one or more three-dimensional features (e.g., wherein smoothness orroughness is based on a spatial deviation from a mean line for mappedpoints of a particular feature). In some embodiments, the respectiveidentified differences include a respective opacity of the one or morethree-dimensional features. In some embodiments, the respectiveidentified differences include a respective color gradient of the one ormore three-dimensional features. In some embodiments, the respectiveidentified differences include a respective internal structure of theone or more three-dimensional features.

In some embodiments, an alert is generated (548) with respect to thetarget (and is optionally displayed on the processing device or otherdisplay device) when the characteristic satisfies a first changethreshold, wherein the target is associated with the first changethreshold. Change thresholds may be quantitative thresholds (e.g.,percentage, numerical quantity, etc. of change, rate of change,differential rates of change, etc.) or qualitative thresholds (e.g.,transition from a specific color pigmentation, shape, etc. to another).Change thresholds may also be defined with respect to any predefinedperiod of time (e.g., over 10 days, a month, etc.). For example, anobserved lesion growth rate in excess of 10% over the course of a monthmay trigger the generation and display of an alert to a target subjector a physician.

For situations in which the systems discussed above collect informationabout users, the users may be provided with an opportunity to opt in/outof programs or features that may collect personal information (e.g.,information about a user's preferences or a user's contributions tosocial content providers). In addition, in some embodiments, certaindata may be anonymized in one or more ways before it is stored or used,so that personally identifiable information is removed. For example, auser's identity may be anonymized so that the personally identifiableinformation cannot be determined for or associated with the user, and sothat user preferences or user interactions are generalized (for example,generalized based on user demographics) rather than associated with aparticular user.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages which are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beapparent to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the embodimentswith various modifications as are suited to the particular usescontemplated.

What is claimed is:
 1. A computer-implemented method of aggregatingmulti-dimensional data for a target at a central controlling system, themethod comprising: constructing a composite map based on a previousdataset, wherein the previous dataset includes: a plurality oftwo-dimensional pixilated images of the target obtained by one or morecomputer-enabled imaging devices during a first time, and respectivemeta data associated with each of the plurality of two-dimensionalpixilated images, wherein the respective meta data indicates (i)positional and orientation data that includes a position and orientationof a respective computer-enabled imaging device when the respectiveimage is obtained, (ii) an indication of when the respective image isobtained, and an (iii) identity of the respective computer-enabledimaging device; using each of the one or more computer-enabled imagingdevices to collect respective image data of the target during a secondtime subsequent to the first time, by causing each respectivecomputer-enabled imaging device of the one or more computer-enabledimaging devices to execute a method of data capture, including: inaccordance with the composite map based on the previous dataset,obtaining a respective two-dimensional pixilated image of the target;associating meta data with the respective two-dimensional pixilatedimage, wherein the respective meta data indicates respective (i)positional and orientation data that includes a position and orientationof the respective computer-enabled imaging device when the respectiveimage is obtained, (ii) an indication of when the respective image isobtained, and an (iii) identity of the respective computer-enabledimaging device; and communicating to the central controlling system therespective two-dimensional pixilated image of the target and therespective meta data; at the central controlling system, the centralcontrolling system having one or more processors and memory for storingone or more programs for execution by the one or more processors,executing the method of: receiving one or more respectivetwo-dimensional pixilated images and associated meta data collected fromthe second time from each of the one or more computer-enabled imagingdevices; using the position and orientation of each of the respectivecomputer-enabled imaging devices when the respective images wereobtained, and the indication of when the images were obtained, to indexthe images against the target, thereby aggregating multi-dimensionaldata for the target; extracting, from the aggregated multi-dimensionaldata for the target, temporal information about a characteristic of thetarget over time; and identifying a deficiency in the composite map,wherein the obtaining, for each respective computer-enabled imagingdevice, of the respective two-dimensional pixilated image is responsiveto and in accordance with the identified deficiency.
 2. Thecomputer-implemented method of claim 1, further comprising identifying atwo-dimensional feature of the target using the composite map, whereinthe obtaining, for each respective computer-enabled imaging device, ofthe respective two-dimensional pixilated image is in accordance with theidentified two-dimensional feature.
 3. The computer-implemented methodof claim 1, further comprising identifying a three-dimensional featureof the target using the composite map, wherein the obtaining, for eachrespective computer-enabled imaging device, of the respectivetwo-dimensional pixilated image is in accordance with the identifiedthree-dimensional feature.
 4. The computer-implemented method of claim1, wherein obtaining, for each respective computer-enabled imagingdevice, the respective two-dimensional pixilated image in accordancewith the composite map comprises: obtaining the respectivetwo-dimensional pixilated image in accordance with the positional andorientation data associated with the plurality of two-dimensionalpixilated images of the previous dataset.
 5. The computer-implementedmethod of claim 1, wherein the position and orientation of therespective computer-enabled imaging device, when the respectivetwo-dimensional pixilated image is obtained, are substantially alignedwith the positional and orientation data associated with at least one ofthe plurality of two-dimensional pixilated images from the previousdataset.
 6. The computer-implemented method of claim 1, wherein: theimage data collected by each of the one or more computer-enabled imagingdevices comprises a plurality of two-dimensional pixilated imagesobtained during the second time; and a position and orientation of arespective one of the one or more computer-enabled imaging devices, foreach of the plurality of two-dimensional pixilated images obtainedduring the second time, are substantially aligned with the positionaland orientation data associated with a respective one of the pluralityof two-dimensional pixilated images from the previous dataset.
 7. Thecomputer-implemented method of claim 1, wherein using each respectivecomputer-enabled imaging device to collect respective image data furthercomprises: providing information indicating a proximity of a currentposition and orientation of a respective computer-enabled imagingdevice, to a respective position and orientation associated with atleast one of the plurality of two-dimensional pixilated images from theprevious dataset.
 8. The computer-implemented method of claim 7, whereinproviding information indicating the proximity comprises displaying, onthe respective computer-enabled imaging device, visual informationindicating the proximity so as to enable alignment of the respectiveposition and orientation of the respective computer-enabled imagingdevice to match the position and orientation associated with the atleast one of the plurality of two-dimensional pixilated images from theprevious dataset.
 9. The computer-implemented method of claim 1, furthercomprising: using the composite map, spatially registering the one ormore respective two-dimensional pixilated images obtained during thesecond time.
 10. The computer-implemented method of claim 9, whereinconstructing the composite map comprises: extracting two-dimensionalfeatures from each image of the previous dataset; and using thetwo-dimensional features extracted from the previous dataset, creating athree-dimensional composite map comprising a plurality ofthree-dimensional points, wherein each respective three-dimensionalpoint of the plurality of three-dimensional points is associated withone of a plurality of respective sets of the two-dimensional featuresextracted from the previous dataset, each of the respective setsrepresenting an appearance of an associated respective three-dimensionalpoint across the previous dataset.
 11. The computer-implemented methodof claim 10, wherein the spatially registering comprises: extractingtwo-dimensional features from each image obtained during the secondtime; and matching a set of the two-dimensional features extracted fromthe images obtained during the second time, with one of the plurality ofrespective sets of the two-dimensional features extracted from theprevious dataset, thereby associating the set of the two-dimensionalfeatures extracted from the images obtained during the second time witha respective three-dimensional point, of the three-dimensional compositemap, that is associated with the matched set of the two-dimensionalfeatures extracted from the previous dataset.
 12. Thecomputer-implemented method of claim 1, wherein constructing thecomposite map comprises: matching a two-dimensional feature in a firstimage and in a second image of the plurality of two-dimensionalpixilated images; estimating a parallax between the first image and thesecond image using respective meta data associated with the first imageand the second image; and adding, when the parallax between the firstimage and the second image satisfies a parallax threshold, a two orthree-dimensional point to the composite map at a distance obtained bytriangulating the first image and the second image using respective metadata associated with the first image and the second image.
 13. Thecomputer-implemented method of claim 1, wherein: the composite map basedon the previous dataset is a first composite map, and the image datacollected during the second time by using each of the one or morecomputer-enabled imaging devices comprises a plurality oftwo-dimensional pixilated images of the target and the associated metadata, the method further comprising: constructing a second composite mapbased on the image data collected during the second time; and comparingthe second composite map with the first composite map to identifyrespective differences in one or more three-dimensional features fromthe first time to the second time.
 14. The computer-implemented methodof claim 13, wherein the respective identified differences include arespective shape of the one or more three-dimensional features.
 15. Thecomputer-implemented method of claim 13, wherein the respectiveidentified differences include a respective volume of the one or morethree-dimensional features.
 16. The computer-implemented method of claim13, wherein the respective identified differences include a respectivesurface contour of the one or more three-dimensional features.
 17. Thecomputer-implemented method of claim 13, wherein the respectiveidentified differences include a respective smoothness or roughness ofthe one or more three-dimensional features.
 18. The computer-implementedmethod of claim 13, wherein the respective identified differencesinclude a respective opacity of the one or more three-dimensionalfeatures.
 19. The computer-implemented method of claim 13, wherein therespective identified differences include a respective color gradient ofthe one or more three-dimensional features.